Data segmentation using masks

ABSTRACT

A vehicle can include various sensors to detect objects in an environment. Sensor data can be captured by a perception system in a vehicle and represented in a voxel space. Operations may include analyzing the data from a top-down perspective. From this perspective, techniques can associate and generate masks that represent objects in the voxel space. Through manipulation of the regions of the masks, the sensor data and/or voxels associated with the masks can be clustered or otherwise grouped to segment data associated with the objects.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application which claims priority to commonlyassigned, co-pending U.S. patent application Ser. No. 15/963,833, filedApr. 26, 2018. Application Ser. No. 15/963,833 is fully incorporatedherein by reference.

BACKGROUND

Autonomous vehicles can use sensors to capture data of an environment.To navigate an environment effectively, autonomous vehicles use sensordata to detect objects in the environment to avoid collisions.Segmentation techniques can be used for associating sensor data withobjects. Segmentation of three-dimensional data, however, presentsparticular challenges that can result in inaccurate object detection,segmentation, and/or poor responsiveness of object detection andsegmentation.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 illustrates a pictorial flow diagram of an example process fortop-down segmentation.

FIG. 2 depicts an example vehicle capturing sensor data of anenvironment.

FIG. 3A depicts an example voxel space with sensor data associated withthe voxel space.

FIG. 3B depicts an example portion of a voxel space viewed from atop-down perspective with sensor data associated with the portion of thevoxel space.

FIG. 3C depicts an example voxel space viewed from a top-downperspective with a mask associated with the portion of the voxel space.

FIG. 4A depicts a pictorial flow diagram of an example process ofgenerating an expanded mask.

FIG. 4B depicts a pictorial flow diagram of an example process ofgenerating masks associated with different objects represented in thevoxel space.

FIG. 5 depicts an example process for expanding a first mask to generatea second mask.

FIG. 6 depicts a block diagram of an example system for implementing thetechniques described herein.

FIG. 7 depicts an example process for training a machine learningalgorithm to output one or more masks.

FIG. 8 depicts an example process for commanding an autonomous vehiclebased at least in part on sensor data.

DETAILED DESCRIPTION

This disclosure describes methods, apparatuses, and systems forperforming segmentation on three-dimensional data represented in avolume element (“voxel”) space to determine objects in an environment.For example, an autonomous vehicle can include a plurality of sensors tocapture sensor data corresponding to an environment of the vehicle. Thesensor data can include data associated with an environment where, insome instances, multiple objects (e.g., pedestrians, vehicles,bicyclists, etc.) are located adjacent or in close proximity to oneanother. The techniques described herein can include, but are notlimited to, capturing the sensor data, associating the sensor data witha voxel space, viewing the sensor data from a “top-down perspective,”and using a machine-learning algorithm to output masks associated withthe one or more objects. Such algorithms may be trained to output masksrepresenting an region smaller than the detected one or more objects sothat, in an inference mode, a size of the masks can be increased until athreshold is reached, where the threshold can be represented by a fixedvalue margin or a percentage of the mask, or until the mask intersectsanother mask representing another vehicle and/or object. The voxelswithin the masks can be associated with one or more object identifiersand the data associated with the masks can be segmented. The segmenteddata can be passed to other classifiers to classify the one or moreobjects and/or to a planning system to determine a trajectory of thevehicle. The segmentation techniques can be used on LIDAR data, RADARdata, image data, etc.

As discussed herein, sensor data can be captured by one or more sensorsand can be represented in a voxel space. The voxel space can includerepresenting the sensor data in a grid of voxels in a three-dimensionalspace. With the sensor data represented in the voxel space, processingcan include using a machine learning algorithm to output a maskassociated with an object in the environment. In some instances, themask can be represented in a plan view or top-down perspective. In otherinstances, the mask can be represented in an elevation view or a sideperspective. As mentioned above, such a machine-learning algorithm maybe trained to output a mask having dimensions slightly smaller thanassociated ground truth objects.

By expanding a first mask output during inference, the system cangenerate a second mask (e.g., also referred to as an “expanded mask”)associated with the object in the environment. In some instances, a sizeof the mask can be increased up to a threshold increase, andaccordingly, the second mask can represent a portion of the sensor datawith an overall higher confidence level of representing an object thanwithout the second mask. Additionally, in some instances, the secondmask can represent a portion of the sensor data with an overall higherconfidence level of representing an object than a mask without thetechniques described herein.

The system can segment the sensor data based at least in part on thesecond mask. Based on the segmentation data, the system can detect anddetermine a number of objects associated with the sensor data and theenvironment. In some instances, the system can use the segmentation datato generate a trajectory for a vehicle and control the vehicle totraverse an environment.

As can be understood, determining and detecting objects in anenvironment can require a significant amount of cost in terms of theamount of data processing required and computing resources.Additionally, transmitting significant amounts of data increases powerconsumption and may also increase network congestion. Traditionalmethods may also consume a significant amount of time to determine anddetect objects in an environment. Meanwhile, methods of determining anddetecting objects in an environment must provide accurate results,particularly in the context of vehicles and autonomous vehicles. Forexample, previous segmentation techniques have used clusteringtechniques or region growing techniques which can result in combiningdiscrete objects as a single object (e.g., undersegmentation), as wellas techniques which divide a single object into multiple discreteobjects (e.g., oversegmentation). Thus, the techniques discussed hereinimprove the accuracy of segmentation on various types of data.

The top-down segmentation techniques discussed herein can improve afunctioning of a computing device by reducing the amount of computingresources to segment sensor data. In some instances, reducing the amountof computing resources takes the form of reducing the amount of dataoperated on. In other instances, reducing the amount of computingresources takes the form of more efficiently processing the data. Inother instances, reducing the amount of computing resources takes theform of both reducing the amount of data used as wells as moreefficiently processing the data. Additionally, in some instances,reducing the amount of computing resources includes reducing processingtime, reducing memory usage, reducing network congestion, and/orreducing power consumption. The techniques described herein can alsomaintain and/or increase the accuracy of determining and detectingobjects in an environment. As can be understood, maintaining and/orincreasing the accuracy of determining and detecting objects can improvesafety outcomes, particularly in the context of vehicles and autonomousvehicles. The techniques described herein can also improve safetyoutcomes by reducing the processing time of determining and detectingobjects. As can be understood, in the context of vehicles and autonomousvehicles, situations may arise where determination and detection ofobjects must occur quickly and accurately. As a non-limiting example,other methods which undersegment and/or oversegment objects in anenvironment may cause errors in subsequent systems (e.g., classificationsystems, planning systems, etc.) which rely on such data for safevehicle operation. Thus, the techniques described herein can improve afunctioning of a computing device as well as improve safety outcomes.

The techniques described herein can be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Although discussed in the context of an autonomousvehicle, the techniques described herein can be applied to a variety ofsystems requiring determination and detection of objects, and is notlimited to autonomous vehicles. In another example, the methods,apparatuses, and systems may be used in an aviation or nautical context.Additionally, the techniques described herein may be used with real data(e.g., captured using sensor(s)), simulated data (e.g., generated by asimulator), or any combination of the two.

FIG. 1 illustrates a pictorial flow diagram of an example process 100for capturing sensor data, associating the sensor data with a volumeelement (“voxel”) space, receiving a mask representing an object in thevoxel space, expanding the mask to generate an expanded mask or a secondmask, and segmenting, based at least in part on the second mask, thesensor data.

At operation 102, the process 100 can include capturing LIDAR (lightdetection and ranging) data as the sensor data. The operation 102 canalso include capturing a variety of sensor data which can include RADARdata, SONAR data, image data, audio data, infrared data, location data,or any combination thereof. In some instances, the operation 102 caninclude capturing sensor data from a plurality of sensors. In someinstances, operation 102 can include extracting a portion of the sensordata for processing, such as over a period of time. An example of sensordata is illustrated in an example 104, which may include LIDAR data(e.g., data points or a point cloud) associated with various objects inan environment, such as vehicles, roads, trees, pedestrians, buildings,etc.

At operation 106, the process 100 can include associating the sensordata with a voxel space. An example 108 illustrates a voxel spaceincluding five voxels in each dimension (e.g., x, y, and z), althoughany number of voxels may be included in the voxel space. In someinstances, the voxel space may correspond to a physical environment,such as an area around an origin or a virtual origin of the sensor data.For example, the voxel space may represent an area 100 meters wide, 100meters long, and 20 meters high. Further, each voxel (e.g., a voxel 110)in the voxel space may represent a physical area, such as 25 centimetersin each dimension. As can be understood in the context of thisdisclosure, the voxel space may represent any region of an environment(e.g., a region proximate to the vehicle which is updated as the vehiclemoves), and individual voxels may represent any volume as well. In someinstances, voxels may be a uniform size throughout the voxel space,while in some instances, a volume of a voxel may vary based on alocation of the voxel relative to an origin of the data. For example, asthe density of sensor data may decrease as a distance from a sensorincreases, the size of a voxel in a voxel space may increase inproportion to a distance from the voxel to a sensor (or an originrepresenting a plurality of sensors).

In one example, the voxel space may be initialized as empty space andsensor data may be added to the voxel space as it is captured. Inanother example, the voxel space may be initialized with datarepresenting a global map of previously captured data. In the case ofusing global map data, the operations may include comparing the locallycaptured sensor data with the global data to localize the vehicle in theglobal map space.

In some instances, the operation 106 can include mapping individual datapoints of the sensor data to individual voxels. In some instances, theoperation 106 can include subtracting a motion vector associated withthe sensor data, for example, in an event that the sensor data iscaptured by a moving platform, such as an autonomous vehicle, to convertthe sensor data to, for example, a stationary reference point. That is,in some instances (e.g., when using RADAR data), the sensor data may beassociated with a voxel space that is fixed with respect to a globalmap, for example (e.g., in contrast to a voxel space fixed with respectto a moving vehicle). In some instances, the operation 106 can includediscarding or omitting voxels that do not include data, or that includea number of data points below a threshold number, in order to create asparse voxel space. Further, in some instances, the operation 106 caninclude aligning a pose of the vehicle (e.g., an orientation of thevehicle) and associated sensor data with the voxel space, for example,to compensate or adjust for possible errors associated with a positionof the vehicle with respect to the voxel space.

Further, in some instances, the operation 106 can include statisticallyaccumulating sensor data and processing the sensor data as it is addedto individual voxels. For example, individual voxels may include sensordata representing a number of data points, an average intensity, anaverage x-value of the sensor data, an average y-value of the sensordata, an average z-value of the sensor data, and/or a covariance matrixbased on the sensor data associated with the individual voxel. Thus, insome instances, data associated with individual voxels may representprocessed data, in part, to improve processing performance of thesystem.

At operation 112, the process 100 can include receiving a first maskrepresenting an object in the voxel space. In some instances, a machinelearning algorithm can be trained to output at least one mask associatedwith a voxel space. Additionally, in some instances, the machinelearning algorithm can use sensor data to output the mask. In otherinstances, the machine learning algorithm can use sensor data and/orprocessed sensor data to output the mask. In the context of thisapplication, “top-down” refers to the corresponding voxel data as inputinto such a machine learning algorithm. Particularly, the algorithmaccepts voxel data in the form of an image of size equal to the number Xand Y discretizations of the grid, the image having a number of channelsequal to the number of discretizations in the Z dimension, where the Zdimension is indicative of a height dimension, as well as any othernumber of channels (such as additional features) as described herein.That is, the voxel data is effectively rotated such that it is viewedfrom above and subsequently input into a machine learning algorithm. Insome examples, the algorithm can accept the voxel data that iseffectively rotated such that it is viewed from an elevation view or aside view.

The mask, in some instances, can represent a region in a voxel spaceassociated with an object. As will be described in detail below, such amachine learning algorithm is trained to output a mask smaller indimension than the associated object. The process 100, in otherinstances, can include generating a mask associated with an object inthe voxel space. In both instances, either receiving the mask orgenerating the mask, the mask can be based at least in part on thesensor data captured in the operation 102.

An example 114 illustrates a top-down view or a plan view of voxel space116. The example 114 also illustrates sensor data 118. For purposes ofillustration, the sensor data 118 can be the sensor data captured at theoperation 102 associated with an environment. The example 114 alsoillustrates a first mask 120 associated with the voxel space 116. Thefirst mask 120 can represent an object in the voxel space and can beassociated with an object as shown in example 104. For purposes ofillustration and as a non-limiting example, the first mask 120 isassociated with a smaller number of voxels in the voxel space 116 thanthe sensor data 118.

In some instances, individual voxels of the voxels space 116 cancomprise a plurality of features. As a non-limiting example, individualvoxels can comprise features such as mean or average three-dimensionalcoordinate values, a covariance matrix of the coordinate values,semantic segmentation probabilities (that is, a set of probabilitiesindicating whether the voxel belongs to different classes of objects),an indication of occupancy (i.e., whether there is any data in the voxelor not), ray casting information (e.g., indicating whether a ray from aLIDAR has passed through a voxel), and/or background subtractioninformation (e.g., indicating whether an object represented in sensordata is a background object (e.g., a static object) or whether an objectis a transitory object (e.g., parked car, pedestrian, animal, car,etc.)). In some instances, the voxel space 116 can represent a“collapsed” version of the full voxel space as shown in the example 108.In other instances, the voxel space 116 can be represented or envisionedas a plurality of pseudo-pixels where each pseudo-pixel is associatedwith a column of voxels. Additionally, in some instances, individualpseudo-pixels can comprise or can be associated with all of the featuresof the voxels associated with the individual pseudo-pixel. As anon-limiting example, a voxel grid of dimensions 400×400×7 may berepresented as a 400×400 image with 119 channels. That is, the 119channels can represent 7 channels for each of the voxel grid heighttimes 17 features per height voxel (three channels of average position,nine channels for the covariance matrix, an occupancy channel, and fourchannels of semantic classification probabilities).

At operation 122, the process 100 can include expanding the first maskto generate a second mask. In some instances, the system can expand thefirst mask by a margin, either a fixed value or a percentage, or both.In other instances, the system can expand the first mask by a fixedvalue or a percentage, or both, incrementally. In some instances, thesystem can expand the first mask based at least in part on a regiongrowing operation associated with an occupancy of voxels orpseudo-pixels. In some instances, the second mask can represent the samearea as the first mask. Additionally, in some instances, an expansion ofthe first mask may not be uniform in all directions. Thus, for purposesof illustration, when generating the second mask, the system can expandone side of the first mask by a greater or lesser amount than adifferent side of the first mask.

In other instances, the second mask, as a result of the operationsdiscussed herein, can represent a smaller area or a larger area relativeto the first mask. Additionally, the system can generate the second maskusing at least a portion of the sensor data. In other instances, thesystem can generate the second mask using margin data associated withthe first mask from the machine learning algorithm. In other instances,the system can generate the second mask while referencing an additionalmask representing an additional object in the voxel space. Thus, forpurposes of illustration, the system can expand the first mask until itreaches threshold increase and/or until it reaches an additional mask.

An example 124 illustrates a top-down view of voxel space 116. Theexample 124 also illustrates the first mask 120 as discussed in theexample 114. The example 124 additionally illustrates an expanded maskor a second mask 126 as discussed above. Thus, for purposes ofillustration, the second mask 126 can represent the first mask 120 afterexpanding the first mask 120 by a margin or by using clusteringtechniques based at least in part on region growing algorithms.

At operation 128, the process 100 can include segmenting, based at leastin part on the second mask, the sensor data. In some instances, thesecond mask can be associated with an object in the voxel space.Additionally, in some instances, the second mask can be associated witha portion of the sensor data. Thus, for purposes of illustration, thesystem can segment the sensor data by segmenting the sensor dataassociated with the second mask directly. Additionally, the system cansegment the sensor data associated with the voxels, where the voxels canbe associated with the second mask.

For purposes of illustration, the system can apply one or moreclustering techniques, such as region growing, to the sensor data or thevoxels associated with the second mask to associate all nearby data witha single object. Additionally, the system can associate an objectidentifier with the sensor data or the voxels associated with the secondmask. As above, such region growing may propagate until there are noadditional occupied voxels within a given radius, for a preset number ofvoxels, or until the region collides with one or more additional regionsgrown substantially simultaneously.

The segmentation techniques described herein can improve a functioningof a computing device by providing a framework for efficientlysegmenting data for object detection. For example, by using a top-downperspective, the amount of data can be reduced, which as describedabove, can reduce the amount of computing resources consumed.Additionally, reducing the amount of data can allow a computing deviceto process the data more efficiently. In some instances, the system cantransmit some of the data to a different computing device and,therefore, reduce network congestion by transmitting less data. In someinstances, the system can also reduce the amount of processing time tosegment data for object detection. In some instances, a boundary of thesegmentation can be defined at least in part on the second mask. Thatis, in some cases, without the second mask, a clustering technique maycluster voxels from multiple discrete objects into a single object.Thus, the use of the techniques discussed herein can improve an accuracyof segmentation by preventing merging of regions associated withdiscrete objects.

In some instances, data associated with the detected objects can beprovided to a planner system to generate a trajectory for an autonomousvehicle that traverses an environment using the trajectory. In someinstances, data associated with the detected objects can be provided toother segmentation algorithms and/or classification algorithms forfurther processing. These and other operations are possible as part ofthe techniques described herein.

FIG. 2 illustrates an example environment 200 through which an examplevehicle 202 is traveling. The example vehicle 202 may be a driverlessvehicle, such as an autonomous vehicle configured to operate accordingto a Level 5 classification issued by the U.S. National Highway TrafficSafety Administration, which describes a vehicle capable of performingall safety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In suchexamples, because the vehicle 202 may be configured to control allfunctions from start to completion of the trip, including all parkingfunctions, it may not include a driver and/or controls for driving thevehicle 202, such as a steering wheel, an acceleration pedal, and/or abrake pedal. This is merely an example, and the systems and methodsdescribed herein may be incorporated into any ground-borne, airborne, orwaterborne vehicle, including those ranging from vehicles that need tobe manually controlled by a driver at all times, to those that arepartially or fully autonomously controlled. In some instances, thetechniques can be implemented in any system using machine vision, and isnot limited to vehicles.

The example vehicle 202 may be any configuration of vehicle, such as,for example, a van, a sport utility vehicle, a cross-over vehicle, atruck, a bus, an agricultural vehicle, and a construction vehicle. Thevehicle 202 may be powered by one or more internal combustion engines,one or more electric motors, hydrogen power, any combination thereof,and/or any other suitable power sources. Although the example vehiclehas four wheels, the systems and methods described herein may beincorporated into vehicles having fewer or a greater number of wheels,tires, and/or tracks. The example vehicle 202 may have four-wheelsteering and may operate generally with equal performancecharacteristics in all directions, for example, such that a first end ofthe vehicle 202 is the front end of the vehicle 202 when traveling in afirst direction, and such that the first end becomes the rear end of thevehicle 202 when traveling in the opposite direction. Similarly, asecond end of the vehicle 202 is the front end of the vehicle whentraveling in the second direction, and such that the second end becomesthe rear end of the vehicle 202 when traveling in the oppositedirection. These example characteristics may facilitate greatermaneuverability, for example, in small spaces or crowded environments,such as parking lots and/or urban areas.

A vehicle such as the example vehicle 202 may be used to travel throughthe environment 200 and collect data. For example, the vehicle 202 caninclude one or more sensors where, for purposes of illustration, the oneor more sensors can be one or more LIDAR sensors. Other sensor types arecontemplated (e.g., RADAR sensors, SONAR sensors, image sensors, audiosensors, infrared sensors, location sensors, etc., or any combinationthereof).

As the vehicle 202 travels through the environment 200, the sensors cancapture sensor data associated with the environment. Additionally, someof the sensor data can be associated with objects (e.g., trees 204,vehicles 206, and pedestrians 208). The sensor data can be associatedwith other objects including, but not limited to, buildings, roadsurfaces, signage, barriers, etc.

FIG. 3A illustrates an example 300 including a three-dimensional voxelspace 302 with sensor data 304 associated with the voxel space 302. Asthe example 300 illustrates, the voxel space 302 can extend in the threedimensions, and the example 300 illustrations a portion of the entirevoxel space. The sensor data 304 can include LIDAR data. In someinstances, the voxel space 302 can represent any number of features,including but not limited to, a covariance matrix, location information,classification information, segmentation information, a number ofobservations, whether the voxel is occupied or not, at the like. Thoughdepicted as a multiple data points for illustrative purposes, suchsensor data 304 is stored per voxel.

FIG. 3B illustrates an example 306 of the voxel space 302 from atop-down perspective or plan view and illustrates the sensor data 304associated with the voxel space 302.

FIG. 3C illustrates an example 308 of the voxel space 302 and the sensordata 304 associated with the voxel space 302. The example 308 alsoillustrates a mask 310 (which may correspond to the first mask 120 asillustrated in FIG. 1) from a top-down perspective. As discussed above,a machine learning algorithm can output the mask 310. Thus, for purposesof illustration, a machine learning algorithm can use the sensor data304 to output the mask 310 which can, in some instances, represent anobject in the voxel space 302. In other instances, the system cangenerate the mask 310. As discussed above, in some instances, themachine learning algorithm can output the mask 310 that is associatedwith a fewer number of voxels than the sensor data 304. In otherinstances, the machine learning algorithm can output the mask 310 thatis associated with an equal number or a greater number of voxels as thesensor data 304.

FIG. 4A illustrates an example 400 of a voxel space 402 with sensor data404. The example 400 also illustrates a mask 406 from a top-downperspective. For purposes of illustration, the voxel space 402 canrepresent the voxel space 302 and the mask 406 can represent the mask310. As discussed above, a machine learning algorithm can output themask 406 that is associated with a fewer number of voxels than thesensor data 404.

FIG. 4A also illustrates an example 408 of the voxel space 402 and themask 406. Additionally, the example 408 also illustrates an expandedmask 410. As a non-limiting example, by performing the techniquesdescribed herein, the example 400 can result in the example 408. In someinstances, the system can use clustering techniques or region growingtechniques to combine discrete voxels to generate the expanded mask 410.In other instances, the system can generate the expanded mask 410 byincreasing the mask 406 by a margin, where the margin can be a fixedvalue or percentage. In other instances (and as illustrated in FIG. 4B),the system can generate the expanded mask 410 until the expanded mask410 intersects a different mask. Further, in other instances, the systemcan use a combination of the above techniques to generate the expandedmask 410. The example 408 shows an instance where, by performing thetechniques described herein, the expanded mask 410 is associated withthe voxels in the voxels space 402 that were also associated with thesensor data 404. In other instances, the expanded mask 410 can beassociated with fewer than or greater than the number of voxels thatwere associated with the sensor data 404.

FIG. 4B illustrates an example 414 of a voxel space 416 with sensor data418. The example 414 also illustrates a mask A 420 and a mask B 422. Ascan be understood, FIG. 4A represents an environment where one object(e.g., a vehicle) is present, while FIG. 4B represents an environmentwhere two objects (e.g., two vehicles) can be adjacent or in closeproximity to one another in the environment. As discussed above, amachine learning algorithm can output mask A 420 and mask B 422 asassociated with a fewer number of voxels than the sensor data 418.

FIG. 4B also illustrates an example 424 of the voxel space 416 with themask A 420 and the mask B 422. As a non-limiting example, by performingthe techniques described herein, the example 414 can result in theexample 424. The example 424 also shows an expanded mask A 424 and anexpanded mask B 426. As the system generates the expanded masks, thesystem can determine that either the expanded mask A 424 or the expandedmask B 426, or both, can meet or intersect with another mask. In someinstances, the system can expand the first masks by a margin that is afixed value or a percentage. Then, the system can reduce the expandedmasks incrementally until they do not satisfy an intersection criterion.In other instances, the system can expand the masks incrementally whilechecking for either meeting or intersecting with another mask. In theexample 424, the system expanded the mask A 420 and the mask B 422 bymargins such that the resulting expanded mask A 424 and expanded mask B426 do not intersect. In some instances, the system can be configured toallow the expanded masks to meet and/or intersect by an overlapthreshold. In other instances, the system can be configured to expandthe masks but not allow the expanded masks to meet. For example, upondetermining that masks intersect or touch, the system can reduce a sizeof one or both mask to introduce a buffer between the two masks.

After generating the expanded mask(s) or the second mask(s), the systemcan, as discussed above, use clustering techniques such as regiongrowing on the sensor data or the voxels associated with the secondmask(s) to determine an object in the voxel space. That is, clusteringoperations can be effectively bounded by a mask. Additionally, thesystem can associate an object identifier with the sensor data or thevoxels associated with the second mask(s).

FIG. 5 depicts an example process 500 for generating an expanded mask(e.g., a second mask). In some examples, some or all of the process 500can be performed by one or more components in FIG. 6, as describedherein.

At operation 502, the process 500 can include capturing sensor data ofan environment. As discussed above, the sensor data can represent LIDARdata, RADAR data, image data, and the like. At operation 504, theprocess 500 can include associating the sensor data with a voxel space.As discussed above, in some instances, sensor data may be represented inthe voxel space as raw sensor data (e.g., with individual <x, y, z,range, time, etc.> values associated with data points) and/or may berepresented as a statistical accumulation of data.

At operation 506, the process 500 can include receiving a first maskrepresenting an object in the voxel space. For example, the first maskmay represent an object such as a vehicle in an environment. In someinstances, the first mask may represent the object from a secondperspective that is different than the first perspective associated withcapturing the sensor data. For example, the first mask may represent atop-down view of the object represented in the voxel space. In someinstances, the first mask can be received from a machine learningalgorithm trained to output a mask associated with voxel data. In someinstances, a mask output by the machine learning algorithm can be basedat least in part on classification data or segmentation data associatedwith the voxel data. In one example, a mask associated with voxelsassociated with a “pedestrian” classification can be generated at as afixed size, while in another example, a mask associated with voxelsassociated with a “vehicle” classification can be generated based on asize of the voxel data. Masks can be generated based on a variety offactors, and is not limited to the examples discussed herein. In someexamples, the machine learning algorithm is trained to output the maskcentered at a cluster of data (an average, a centroid, a highest densityof data, or the like).

At operation 508, the process 500 can expand the first mask. Asdiscussed above, the process 500 can expand the first mask by a fixedvalue or a percentage, or the process 500 can expand the first maskincrementally. In some instances, the process 500 can expand the firstmask using clustering techniques based at least in part on a regiongrowing algorithm. In other instances, the process 500 can expand thefirst mask until it meets or intersects a different mask. In someinstances, the operation 508 can be based at least in part onclassification data associated with voxels or pseudo-pixels representedby the first mask. That is, in some cases expanding a mask associatedwith a vehicle, for example, can be different than expanding a maskassociated with a pedestrian. For example, a threshold increase can bebased on a classification type.

At operation 510, the process 500 can include determining whether theexpanded first mask comprises or is associated with all object voxelsthat are associated with sensor data. As a non-limiting example, theoperation 510 can result in the affirmative in a scenario where thevoxel space is associated with sensor data representing a single objectand expanding the mask can include all of the voxels that include sensordata within some radius of the mask. As can be understood, the operation510 can be represented as illustrated in FIG. 4A. If the operation 510results in the affirmative, the process 500 can proceed to operation516. If the operation 510 results in the negative, the process 500 canproceed to operation 512 where the process 500 can perform a checkregarding whether the expanded first mask meets or exceeds a margin. Asa non-limiting example, the operation 512 can result in the affirmativeif the process 500 uses a fixed value or a percentage as a margin. Ifthe operation 512 results in the affirmative, the process 500 canproceed to operation 516. If the operation 512 results in the negative,the process 500 can proceed to operation 514 where the process 500 canperform a check regarding whether the expanded first mask meets orintersects with a different mask.

As a non-limiting example, the operation 514 can result in theaffirmative if the process 500 expands the first mask in a voxel spacewhere another mask representing another object is adjacent to or inclose proximity to the first mask. As can be understood, the operation514 can be represented as illustrated in FIG. 4B. If the operation 514results in the negative, the process 500 can return to operation 508. Ifthe operation 514 results in the affirmative, the process 500 canproceed to operation 516. Shown in this configuration for purposes ofillustration, the process 500 can perform the operations, 510, 512, and514 in any order and also in parallel.

Additionally, the operations 510, 512, and 514 can be performedindependently on each side (e.g., left, right, top, bottom) of the firstmask. Thus, as can be understood, the first mask can expand on one sideuntil it intersects with a different mask, expand on a different sideuntil it meets or exceeds a margin, and expand on yet a different sideuntil it comprises all voxels associated with sensor data on that side,for example, by using clustering techniques based at least in part on aregion growing algorithm. Additionally, though depicted in FIG. 5 forillustrative purposes, not all operations 510, 512, 514 need beperformed. As a non-limiting example, any or all of operations 510, 512,or 514 may be omitted.

At operation 516, the process 500, after expanding the first mask, cangenerate a second mask (e.g., an expanded mask). In some instances, thesecond mask can represent an area larger than the first mask. In otherinstances, the second mask can represent an area substantially similarto the first mask. In some instances, in the case where the expandedfirst mask meets or intersects with a different mask, the operation 516can include generating the second mask with a “buffer” between differentmasks, to improve segmentation of the voxel data in other operations,for example.

FIG. 6 depicts a block diagram of an example system 600 for implementingthe techniques discussed herein. In at least one example, the system 600can include a vehicle 602, which can be the same vehicle as the vehicle202 described above with reference to FIG. 2.

The vehicle 602 can include a vehicle computing device 604, one or moresensor systems 606, one or more emitters 608, one or more communicationconnections 610, at least one direct connection 612 (e.g., forphysically coupling with the vehicle 602 to exchange data and/or toprovide power), and one or more drive modules 614.

The one or more sensor systems 606 can be configured to capture sensordata associated with an environment, which can be an environment similarto the environment 200 described above with reference to FIG. 2.

The vehicle computing device 604 can include one or more processors 616and memory 618 communicatively coupled with the one or more processors616. In the illustrated example 600, the vehicle 602 is an autonomousvehicle; however, the vehicle 602 can be any other type of vehicle. Inthe illustrated example, the memory 618 of the vehicle computing device604 stores a localization component 620, a training component 622, aplanning component 624, one or more system controllers 626, and aperception component 628. Though depicted as residing in the memory 618for illustrative purposes, it is contemplated that the localizationcomponent 620, the training component 622, the planning component 624,the one or more system controllers 626, and the perception component 628can additionally, or alternatively, be accessible to the vehicle 602(e.g., stored remotely).

In at least one example, the localization component 620 can includefunctionality to receive data from the sensor system(s) 606 to determinea position of the vehicle 602. For example, the localization component620 can include and/or request/receive a three-dimensional map of anenvironment and can continuously determine a location of the autonomousvehicle within the map. In some instances, the localization component620 can use SLAM (simultaneous localization and mapping) or CLAMS(calibration, localization and mapping, simultaneously) to receive imagedata, LIDAR data, RADAR data, SONAR data, IMU data, GPS data, wheelencoder data, or any combination thereof, and the like to accuratelydetermine a location of the autonomous vehicle. In some instances, thelocalization component 620 can provide data to various components of thevehicle 602 to determine an initial position of an autonomous vehiclefor generating a candidate trajectory, as discussed herein.

In some instances, the training component 622 can include functionalityto receive data from the sensor system(s) 606 and create a trainingdataset for use in a machine learning algorithm to identify classes inthe data. For example, the training component 622 can indicate classessuch as vehicles, pedestrians, bicycles, motorcycles, trees, roadsurfaces, buildings etc. In some instances, the training component 622can include functionality to manually or automatically indicate anextent (e.g., length, width, height) of the objects represented in thesensor data. In some instances, the training data can include athree-dimensional bounding box associated with LIDAR data representingan extent of the objects represented in the training data. The trainingcomponent 622 can then associate sensor data with these classes and/orextents to create a training dataset. In some instances, at least aportion of the training dataset can include ground truth informationrepresenting a mask that is smaller than a true extent (e.g., length andwidth) of an object represented by the object. Such ground truth may becreated, for example, by hand labeled data, by simulated data, or byother algorithms which associate a mask with data, the extents of whichare subsequently reduced by an amount. The training dataset can be usedto train a machine learning algorithm to identify objects within thesensor data. Once the machine learning algorithm is trained, the machinelearning algorithm can then output one or more masks representing one ormore objects based on the sensor data.

In general, the planning component 624 can determine a path for thevehicle 602 to follow to traverse through an environment. For example,the planning component 624 can determine various routes and trajectoriesand various levels of detail. For example, the planning component 624can determine a route to travel from a first location (e.g., a currentlocation) to a second location (e.g., a target location). For thepurpose of this discussion, a route can be a sequence of waypoints fortraveling between two locations. As non-limiting examples, waypointsinclude streets, intersections, global positioning system (GPS)coordinates, etc. Further, the planning component 624 can generate aninstruction for guiding the autonomous vehicle along at least a portionof the route from the first location to the second location. In at leastone example, the planning component 624 can determine how to guide theautonomous vehicle from a first waypoint in the sequence of waypoints toa second waypoint in the sequence of waypoints. In some examples, theinstruction can be a trajectory, or a portion of a trajectory. In someexamples, multiple trajectories can be substantially simultaneouslygenerated (i.e., within technical tolerances) in accordance with areceding horizon technique. A single trajectory of the multipletrajectories in a receding horizon having the highest confidence levelmay be selected to operate the vehicle.

In other examples, the planning component 624 can alternatively, oradditionally, use data from the perception component 628 to determine apath for the vehicle 602 to follow to traverse through an environment.For example, the planning component can receive data from the perceptioncomponent 628 regarding objects associated with an environment. Usingthis data, the planning component 624 can determine a route to travelfrom a first location (e.g., a current location) to a second location(e.g., a target location) to avoid objects in an environment.

In at least one example, the vehicle computing device 604 can includeone or more system controllers 626, which can be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 602. These system controller(s) 626 cancommunicate with and/or control corresponding systems of the drivemodule(s) 614 and/or other components of the vehicle 602, which may beconfigured to operate in accordance with a trajectory provided from theplanning system 624.

In some examples, the perception component 628 can include functionalityto perform object detection, segmentation, and/or classification. Insome examples, the perception component 628 can provide processed sensordata that indicates a presence of an entity that is proximate to thevehicle 602 and/or a classification of the entity as an entity type(e.g., car, pedestrian, cyclist, building, tree, road surface, curb,sidewalk, unknown, etc.). In additional and/or alternative examples, theperception component 628 can provide processed sensor data thatindicates one or more characteristics associated with a detected entityand/or the environment in which the entity is positioned. In someexamples, characteristics associated with an entity can include, but arenot limited to, an x-position (global position), a y-position (globalposition), a z-position (global position), an orientation, an entitytype (e.g., a classification), a velocity of the entity, an extent ofthe entity (size), etc. Characteristics associated with the environmentcan include, but are not limited to, a presence of another entity in theenvironment, a state of another entity in the environment, a time ofday, a day of a week, a season, a weather condition, an indication ofdarkness/light, etc.

In the illustrated example 600, the perception component 628 can includea segmentation component 630. The segmentation component can, in turn,include an association component 632, an expansion component 634, avalidation component 636, and an mask component 638.

In at least one example, the association component 632 can includefunctionality to receive sensor data and associate the sensor data witha voxel space, the voxel space comprising, in some instances, threedimensions. In some instances, the association component 632 can includefunctionality to statistically accumulate sensor data and process thesensor as it is associated with individual voxels. In other instances,the association component 632 can include functionality to associate thesensor data with a voxel space as raw sensor data (e.g., with individual<x, y, z, range, time, etc.> values associated with data points).

In the illustrated example 600, the expansion component 634 can includefunctionality to receive a mask and expand the mask, the maskrepresenting an object in the voxel space, in some examples, in twodimensions. In some instances, the mask representing the object in twodimensions can use a plan view or a top-down perspective. For purposesof illustration, this mask can be referred to as the first mask. Such amask may be provided, for example, by inputting the voxel grid as animage of multiple channels into a machine learning algorithm trained tooutput masks, as described herein. Expansion of the mask may beperformed in accordance with any of the techniques described herein(e.g., as described with respect to FIG. 5).

In the illustrated example 600, the validation component 636 can includefunctionality to determine whether the expansion component 634sufficiently expands the first mask. In some instances, the validationcomponent 636 can determine whether the expansion component 634 expandsthe first mask to comprise all voxels associated with sensor data withina region around the mask (e.g., based at least in part on a seed voxeland/or based at least in part on a distance or radius from a seedvoxel), whether the expansion component 634 expands the first mask tomeet or exceed a margin, or whether the expansion component 634 expandsthe first mask to meet or intersect a different mask. In some instances,the validation component 636 can perform these functions in parallel orserially. Additionally, in some instances, the validation component 636can perform these functions, as the expansion component 634 expands thefirst mask, per side of the first mask. As discussed above, the margincan be a fixed value or a percentage of the first mask. Additionally,the validation component 636 can use techniques such as clusteringtechniques based at least in part on region growing. In some instances,based on the determination of the validation component 636, theexpansion component 634 can continue to expand the first mask. In someinstances, based on the determination of the validation component 636,mask component 638 can receive data from the expansion component 634.

In the illustrated example 600, the mask component 638 can includefunctionality to determine and/or generate a second mask based at leastin part on the data from the expansion component 634. In some instances,the mask component 638 can determine and/or generate the second maskbased at least in part on the first mask.

The segmentation component 630 can include functionality to segment thesensor data associated with the second mask to associate an objectand/or an object identifier with the sensor data associated with thesecond mask. In some instances, the segmentation component 630 can applyone or more clustering techniques, such as region growing, to the sensordata or the voxels associated with the second mask to determine theobject in the voxel space. Additionally, the segmentation component 630can, in some instances, associate an object identifier with the sensordata or the voxels associated with the second mask.

In at least one example, the sensor system(s) 606 can include LIDARsensors, RADAR sensors, ultrasonic transducers, SONAR sensors, locationsensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertialmeasurement units (IMUs), accelerometers, magnetometers, gyroscopes,etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), microphones,wheel encoders, environment sensors (e.g., temperature sensors, humiditysensors, light sensors, pressure sensors, etc.), etc. The sensorsystem(s) 606 can include multiple instances of each of these or othertypes of sensors. For instance, the LIDAR sensors can include individualLIDAR sensors located at the corners, front, back, sides, and/or top ofthe vehicle 602. As another example, the camera sensors can includemultiple cameras disposed at various locations about the exterior and/orinterior of the vehicle 602. The sensor system(s) 606 can provide inputto the vehicle computing device 604. Additionally, and/or alternatively,the sensor system(s) 606 can send sensor data, via the one or morenetworks 640, to the one or more computing device(s) 642 at a particularfrequency, after a lapse of a predetermined period of time, in nearreal-time, etc.

The vehicle 602 can also include one or more emitters 608 for emittinglight and/or sound, as described above. The emitters 608 in this exampleinclude interior audio and visual emitters to communicate withpassengers of the vehicle 602. By way of example and not limitation,interior emitters can include speakers, lights, signs, display screens,touch screens, haptic emitters (e.g., vibration and/or force feedback),mechanical actuators (e.g., seatbelt tensioners, seat positioners,headrest positioners, etc.), and the like. The emitters 608 in thisexample also include exterior emitters. By way of example and notlimitation, the exterior emitters in this example include lights tosignal a direction of travel or other indicator of vehicle action (e.g.,indicator lights, signs, light arrays, etc.), and one or more audioemitters (e.g., speakers, speaker arrays, horns, etc.) to audiblycommunicate with pedestrians or other nearby vehicles, one or more ofwhich may comprise acoustic beam steering technology.

The vehicle 602 can also include one or more communication connection(s)610 that enable communication between the vehicle 602 and one or moreother local or remote computing device(s). For instance, thecommunication connection(s) 610 can facilitate communication with otherlocal computing device(s) on the vehicle 602 and/or the drive module(s)614. Also, the communication connection(s) 610 can allow the vehicle tocommunicate with other nearby computing device(s) (e.g., other nearbyvehicles, traffic signals, etc.). The communications connection(s) 610also enable the vehicle 602 to communicate with a remote teleoperationscomputing device or other remote services.

The communications connection(s) 610 can include physical and/or logicalinterfaces for connecting the vehicle computing device 604 to anothercomputing device or a network, such as network(s) 640. For example, thecommunications connection(s) 610 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as Bluetooth, cellular communication(e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.) or any suitable wired or wirelesscommunications protocol that enables the respective computing device tointerface with the other computing device(s).

In at least one example, the vehicle 602 can include one or more drivemodules 614. In some examples, the vehicle 602 can have a single drivemodule 614. In at least one example, if the vehicle 602 has multipledrive modules 614, individual drive modules 614 can be positioned onopposite ends of the vehicle 602 (e.g., the front and the rear, etc.).In at least one example, the drive module(s) 614 can include one or moresensor systems to detect conditions of the drive module(s) 614 and/orthe surroundings of the vehicle 602. By way of example and notlimitation, the sensor system(s) can include one or more wheel encoders(e.g., rotary encoders) to sense rotation of the wheels of the drivemodules, inertial sensors (e.g., inertial measurement units,accelerometers, gyroscopes, magnetometers, etc.) to measure orientationand acceleration of the drive module, cameras or other image sensors,ultrasonic sensors to acoustically detect objects in the surroundings ofthe drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, suchas the wheel encoders can be unique to the drive module(s) 614. In somecases, the sensor system(s) on the drive module(s) 614 can overlap orsupplement corresponding systems of the vehicle 602 (e.g., sensorsystem(s) 606).

The drive module(s) 614 can include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage junction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive module(s) 614 caninclude a drive module controller which can receive and preprocess datafrom the sensor system(s) and to control operation of the variousvehicle systems. In some examples, the drive module controller caninclude one or more processors and memory communicatively coupled withthe one or more processors. The memory can store one or more modules toperform various functionalities of the drive module(s) 614. Furthermore,the drive module(s) 614 also include one or more communicationconnection(s) that enable communication by the respective drive modulewith one or more other local or remote computing device(s).

In at least one example, the localization component 620 and/or theplanning component 624 can process sensor data, as described above, andcan send their respective outputs, over the one or more network(s) 640,to one or more computing device(s) 642. In at least one example, thelocalization component 620, and/or the planning component 624 can sendtheir respective outputs to the one or more computing device(s) 642 at aparticular frequency, after a lapse of a predetermined period of time,in near real-time, etc.

The vehicle 602 can send sensor data to one or more computing device(s)642, via the network(s) 640. In some examples, the vehicle 602 can sendraw sensor data to the computing device(s) 642. In other examples, thevehicle 602 can send processed sensor data and/or representations ofsensor data to the computing device(s) 642. In some examples, thevehicle 602 can send sensor data to the computing device(s) 642 at aparticular frequency, after a lapse of a predetermined period of time,in near real-time, etc. In some cases, the vehicle 602 can send sensordata (raw or processed) to the computing device(s) 642 as one or morelog files.

The computing device(s) 642 can receive the sensor data (raw orprocessed) and can generate and/or update maps based on the sensor data.Additionally, the computing device(s) 642, in at least one example, caninclude one or more processors 644 and memory 646 communicativelycoupled with the one or more processors 644. In the illustrated example,the memory 646 of the computing device(s) 642 stores the trainingcomponent 648 and the segmentation component 650, which comprises theassociation component 652, the expansion component 654, the validationcomponent 656, and the mask component 658.

The training component 648 can receive the sensor data and create atraining dataset. As can be understood, the training component 648 cancorrespond in functionality discussed in conjunction with the trainingcomponent 622.

In general, the segmentation component 650 can receive data from one ormore vehicles (such as the vehicle 602) and can segment the sensor datato detect objects in an environment. As can be understood, thesegmentation component 650 and components included therein cancorrespond in functionality discussed in connection with thesegmentation component 630.

The processor(s) 616 of the vehicle 602 and the processor(s) 644 of thecomputing device(s) 642 can be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)616 and 644 can comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that can be stored in registers and/ormemory. In some examples, integrated circuits (e.g., ASICs, etc.), gatearrays (e.g., FPGAs, etc.), and other hardware devices can also beconsidered processors in so far as they are configured to implementencoded instructions.

The memory 618 and the memory 646 are examples of non-transitorycomputer-readable media. The memory 618 and the memory 646 can store anoperating system and one or more software applications, instructions,programs, and/or data to implement the methods described herein and thefunctions attributed to the various systems. In various implementations,the memory 618 and the memory 646 can be implemented using any suitablememory technology, such as static random access memory (SRAM),synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or anyother type of memory capable of storing information. The architectures,systems, and individual elements described herein can include many otherlogical, programmatic, and physical components, of which those shown inthe accompanying figures are merely examples that are related to thediscussion herein.

In some instances, aspects of some or all of the components discussedherein can include any models, algorithms, and/or machine learningalgorithms. For example, in some instances, the components in the memory618 and 646 can be implemented as a neural network.

As described herein, an exemplary neural network is a biologicallyinspired algorithm which passes input data through a series of connectedlayers to produce an output. Each layer in a neural network can alsocomprise another neural network, or can comprise any number of layers(whether convolutional or not). As can be understood in the context ofthis disclosure, a neural network can use machine learning, which canrefer to a broad class of such algorithms in which an output isgenerated based on learned parameters.

Although discussed in the context of neural networks, any type ofmachine learning can be used consistent with this disclosure. Forexample, machine learning algorithms can include, but are not limitedto, regression algorithms (e.g., ordinary least squares regression(OLSR), linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridgeregression, least absolute shrinkage and selection operator (LASSO),elastic net, least-angle regression (LARS)), decisions tree algorithms(e.g., classification and regression tree (CART), iterative dichotomiser3 (ID3), Chi-squared automatic interaction detection (CHAID), decisionstump, conditional decision trees), Bayesian algorithms (e.g., naïveBayes, Gaussian naïve Bayes, multinomial naïve Bayes, averageone-dependence estimators (AODE), Bayesian belief network (BNN),Bayesian networks), clustering algorithms (e.g., k-means, k-medians,expectation maximization (EM), hierarchical clustering), associationrule learning algorithms (e.g., perceptron, back-propagation, hopfieldnetwork, Radial Basis Function Network (RBFN)), deep learning algorithms(e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN),Convolutional Neural Network (CNN), Stacked Auto-Encoders),Dimensionality Reduction Algorithms (e.g., Principal Component Analysis(PCA), Principal Component Regression (PCR), Partial Least SquaresRegression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS),Projection Pursuit, Linear Discriminant Analysis (LDA), MixtureDiscriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA),Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g.,Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, StackedGeneralization (blending), Gradient Boosting Machines (GBM), GradientBoosted Regression Trees (GBRT), Random Forest), SVM (support vectormachine), supervised learning, unsupervised learning, semi-supervisedlearning, etc.

Additional examples of architectures include neural networks such asResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.

It should be noted that while FIG. 6 is illustrated as a distributedsystem, in alternative examples, components of the vehicle 602 can beassociated with the computing device(s) 642 and/or components of thecomputing device(s) 642 can be associated with the vehicle 602. That is,the vehicle 602 can perform one or more of the functions associated withthe computing device(s) 642, and vice versa.

FIG. 7 depicts an example process 700 for training a machine learningalgorithm to output one or more masks associated with one or moreobjects. In some examples, some or all of the process 500 can beperformed by one or more components in FIG. 6, as described herein.

At operation 702, the process can include capturing sensor data (e.g.,LIDAR data, image data, etc.) representing at least one object in anenvironment. For example, the environment may include objects including,but not limited to, vehicles, pedestrians, buildings, animals, and thelike.

At operation 704, the process can include generating, based at least inpart on the sensor data, a three-dimensional bounding box associatedwith the at least one object. For example, the bounding box canrepresent an extent (e.g., length, width, height) of the objectrepresented in the image data. In some instances, the operation 704 caninclude associating at least a portion for the sensor data with a voxelspace. In some instances, a voxel of the voxel space can include anynumber of features representing the LIDAR data. For example, features ofa voxel may include, but are not limited to, as mean or averagethree-dimensional coordinate values of the LIDAR data associated withthe voxel, one or more covariance matrices associate one or more degreesof freedom of data (e.g., x-direction, y-direction, z-direction, etc.)semantic segmentation probabilities (e.g., a probability that one ormore LIDAR data points represents one or more objects, including but notlimited to, vehicles, animals, pedestrians, buildings, and the like), anindication of occupancy of the voxel, ray casting information, a freespace indication, background subtraction information, and the like.

At operation 706, the process can include generating, as training dataand based at least in part on the three-dimensional bounding box, a maskrepresenting a region that is smaller than a region associated with theat least on object, the mask associated with a top-down perspective ofthe voxel space. For example, the mask may represent a representationwhich is smaller than the ground truth bounding box of the objectrepresented in the voxel space.

At operation 708, the process can include determining whether there issufficient training data. If no, the process can return to the operation702 to capture additional sensor data and to generate additionalbounding boxes and/or masks. If yes, the process can continue tooperation 710.

At operation 710, the process can include training, based at least inpart on the training data, a machine learning algorithm to output a maskrepresenting a region of an object that that is smaller than the object.For example, the mask may represent the “interior” of the object, suchthat the mask represents relatively high level of confidence that voxelsassociated with the mask are, in fact, associated with the particularobject.

At operation 712, the process can include transmitting the machinelearning algorithm to a system for segmenting captured sensor data. Asdiscussed herein, the system can include an autonomous vehicle or anysystem utilizing machine vision, as discussed herein.

FIG. 8 depicts an example process 800 for commanding an autonomousvehicle based at least in part on a location based on sensor data, asdiscussed herein. For example, some or all of the process 800 can beperformed by one or more components in FIG. 6, as described herein. Forexample, some or all of the process 800 can be performed by the one ormore components of the vehicle 602.

At operation 802, the process can include receiving sensor data capturedby an autonomous vehicle. In some instances, the sensor data can bereceived by the vehicle 602 for determining a region associated with aground surface of the environment.

At operation 804, the process can include generating, based at least inpart on the sensor data, a trajectory for the autonomous vehicle. Forexample, and as discussed herein, the operation 804 can includereceiving segmentation data associated with an object and tracking theobject (e.g., a car, pedestrian, etc.) to determine the trajectory basedat least in part on the segmentation data.

At operation 806, the process can include commanding the autonomousvehicle to follow the trajectory. In some instances, the operation 806can include generating a route, trajectory, and/or control signals toone or more systems of the autonomous vehicle to navigate the autonomousvehicle within the environment.

FIGS. 1, 5, 7, and 8 illustrate example processes in accordance withembodiments of the disclosure. These processes are illustrated aslogical flow graphs, each operation of which represents a sequence ofoperations that can be implemented in hardware, software, or acombination thereof. In the context of software, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular abstract data types. The order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Example Clauses

A. A system comprising: one or more processors; and one or morecomputer-readable media storing instructions executable by the one ormore processors, wherein the instructions, when executed, cause thesystem to perform operations comprising: capturing sensor data using alight detection and ranging (LIDAR) sensor on a vehicle; associating thesensor data with a voxel space, the voxel space comprising at leastthree dimensions; generating a two-dimensional representation of thevoxel space, the two-dimensional representation comprising a number ofimage channels; inputting the two-dimensional representation into amachine learning algorithm; receiving, from the machine learningalgorithm, a first mask representing an object in the voxel space in twodimensions; generating, based at least in part on an expansion of thefirst mask, a second mask, the expansion based at least in part on aregion growing algorithm, a size of the first mask, or an intersectionwith a third mask associated with another object; and segmenting, basedat least in part on the second mask, the sensor data.

B: The system of paragraph A, wherein the machine learning algorithm istrained to receive captured LIDAR data representing a detected objecthaving a first width and a first length and to output a mask associatedwith the detected object, the mask having a second width less than orequal to the first width and a second length less than or equal to thefirst length.

C: The system of paragraph A or B, wherein a pseudo-pixel is associatedwith a voxel of the voxel space, the pseudo-pixel representing a columnof voxels of the voxel space.

D: The system of paragraph C, wherein the pseudo-pixel comprisesfeatures associated with the column of voxels.

E: The system of any of paragraphs A-D, wherein segmenting the sensordata comprises using the region growing algorithm to cluster one or morevoxels of the voxel space within the second mask.

F: A method comprising: capturing sensor data of an environment usingone or more sensors, the sensor data indicative of an object in theenvironment; associating the sensor data with a voxel space; receiving afirst mask associated with a portion of the voxel space, the first maskrepresenting a region smaller in size than the object; generating asecond mask by expanding the first mask; and segmenting, based at leastin part on the second mask, the sensor data.

G: The method of paragraph F, further comprising: generating, based atleast in part on segmenting the sensor data, a trajectory for anautonomous vehicle; and controlling, based at least in part on thetrajectory, the autonomous vehicle to traverse the environment.

H: The method of paragraph F or G, further comprising: inputting atwo-dimensional representation of the voxel space into a machinelearning algorithm; and receiving, as the first mask, an output of themachine learning algorithm.

I: The method of paragraph H, wherein the two-dimensional representationof the voxel space comprises an image having a number of channels basedat least in part on a height of the voxel space and one or morefeatures.

J: The method of paragraph I, wherein the one or more features comprise:an average of sensor data, a number of times sensor data is associatedwith a voxel, a covariance of sensor data, a probability of a voxelbelonging to one or more classifications, a ray casting informationassociated with a voxel; or an occupancy of a voxel.

K: The method of any of paragraphs F-J, wherein the one or more sensorscomprises one or more light detection and ranging (LIDAR) sensors.

L: The method of any of paragraphs F-K, wherein the first mask isgenerated, based at least in part, on classification data associatedwith the sensor data.

M: The method of paragraph L, wherein the classification data is atleast one or more of a vehicle, a bicycle, or a pedestrian.

N: The method of any of paragraphs F-M, further comprising: generatingthe second mask based at least in part on an intersection of anexpansion of the first mask and a third mask associated with anotherobject associated with the voxel space.

O: The method of any of paragraphs F-N, wherein segmenting the sensordata comprises associating one or more voxels of the voxel spaceassociated with the second mask.

P: A non-transitory computer-readable medium storing instructionsexecutable by one or more processors, wherein the instructions, whenexecuted, cause the one or more processors to perform operationscomprising: receiving sensor data of an environment from one or moresensors on an autonomous vehicle, the sensor data associated with anobject in the environment; associating the sensor data with a voxelspace; receiving a first mask associated with a portion of the voxelspace, the first mask representing at least a portion of the objectusing a second perspective; generating a second mask by expanding thefirst mask; and segmenting, based at least in part on the second mask,the sensor data.

Q: The non-transitory computer-readable medium of paragraph P, whereinsegmenting the sensor data comprises associating one or more voxels ofthe voxel space with the second mask.

R: The non-transitory computer-readable medium of paragraph P or Q, theoperations further comprising: generating the second mask based at leastin part on an intersection of an expansion of the first mask and a thirdmask associated with another detected object in the voxel space.

S: The non-transitory computer-readable medium of any of paragraphs P-R,the operations further comprising: inputting, as a two-dimensionalrepresentation, the voxel space into a machine learning algorithm; andreceiving, as the first mask, an output of the machine learningalgorithm, wherein the two-dimensional representation comprises apseudo-image having a length associated with a first dimension of thevoxel space, a width associated with a second dimension of the voxelspace, and a number of channels, and further wherein the number ofchannels is based, at least in part, on a third dimension of the voxelspace and one or more features comprising an average of sensor data, acovariance of sensor data, a number of observations of sensor data, anoccupancy, or one or more probabilities associated with a semanticclassification.

T: The non-transitory computer-readable medium of any of paragraphs P-S,the operations further comprising: generating, based at least in part onsegmenting the sensor data, a trajectory for the autonomous vehicle; andcontrolling, based at least in part on the trajectory, the autonomousvehicle to traverse the environment.

While the example clauses described above are described with respect toone particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses can also beimplemented via a method, device, system, and/or computer-readablemedium.

CONCLUSION

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein can be presentedin a certain order, in some cases the ordering can be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

What is claimed is:
 1. A system comprising: one or more processors; andone or more computer-readable media storing instructions executable bythe one or more processors, wherein the instructions, when executed,cause the system to perform operations comprising: receiving sensor dataassociated with an environment, the sensor data comprising arepresentation of at least a portion of an object in the environment;associating the sensor data with a three-dimensional space; receiving afirst representation associated with a portion of the three-dimensionalspace, the first representation representing at least a portion of theobject using a second perspective; determining a second representationbased at least in part on the first representation; and determining,based at least in part on the second representation, at least a portionof the sensor data associated with the object.
 2. The system of claim 1,wherein: the first representation comprises a first two-dimensionalmask; the second representation comprises a second two-dimensional mask;and the three-dimensional space comprises a voxel space.
 3. The systemof claim 1, the operations further comprising: determining the secondrepresentation based at least in part on the first representation and athird representation associated with another detected object in thethree-dimensional space.
 4. The system of claim 1, wherein: the firstrepresentation is associated with a region smaller in size than theobject; and the second representation is associated with an expansion ofthe first representation.
 5. The system of claim 1, wherein determiningthe at least the portion of the sensor data comprises segmenting thesensor data using a region growing algorithm.
 6. A method comprising:receiving sensor data associated with an environment, the sensor dataindicative of an object in the environment; associating the sensor datawith a multi-channel image; receiving a first representation associatedwith a portion of the multi-channel image, the first representationassociated with a region smaller in size than the object; generating asecond representation based at least in part on the firstrepresentation; and determining, based at least in part on the secondrepresentation, a portion of the sensor data associated with the object.7. The method of claim 6, further comprising: generating, based at leastin part on the portion of the sensor data associated with the object, atrajectory for an autonomous vehicle; and controlling, based at least inpart on the trajectory, the autonomous vehicle to traverse theenvironment.
 8. The method of claim 6, further comprising: inputting themulti-channel image into a machine learning algorithm; and receiving, asthe first representation, an output of the machine learning algorithm.9. The method of claim 8, wherein the multi-channel image comprises anumber of channels based at least in part on a height of a voxel spaceassociated with the multi-channel image and one or more features. 10.The method of claim 9, wherein the one or more features comprise atleast one of: an average of sensor data, a number of times sensor datais associated with a voxel, a covariance of sensor data, a probabilityof a voxel belonging to one or more classifications, a ray castinginformation associated with a voxel; or an occupancy of a voxel.
 11. Themethod of claim 6, wherein the second representation is associated withan expansion of the first representation, wherein the expansion is basedat least in part on a region growing algorithm.
 12. The method of claim6, wherein the first representation is further based at least in part ona classification associated with the object.
 13. The method of claim 12,wherein the classification is at least one or more of a vehicle, abicycle, or a pedestrian.
 14. The method of claim 6, further comprising:generating the second representation based at least in part on anintersection of an expansion of the first representation and a thirdrepresentation associated with another object associated with themulti-channel image.
 15. The method of claim 6, wherein determining theportion of the sensor data associated with the object comprises at leastone of: associating the portion of the sensor data with the secondrepresentation; associating one or more pseudo pixels of themulti-channel image with the second representation; or associating oneor more voxels of a voxel space with the second representation.
 16. Oneor more non-transitory computer-readable media storing instructionsexecutable by one or more processors, wherein the instructions, whenexecuted, cause the one or more processors to perform operationscomprising: receiving sensor data associated with an environment, thesensor data comprising a representation of at least a portion of anobject in the environment; associating the sensor data with athree-dimensional space; receiving a first representation associatedwith a portion of the three-dimensional space, the first representationrepresenting at least a portion of the object using a secondperspective; determining a second representation based at least in parton the first representation; and determining, based at least in part onthe second representation, at least a portion of the sensor dataassociated with the object.
 17. The one or more non-transitorycomputer-readable media of claim 16, wherein: the first representationcomprises a first two-dimensional mask; the second representationcomprises a second two-dimensional mask; and the three-dimensional spacecomprises a voxel space.
 18. The one or more non-transitorycomputer-readable media of claim 16, the operations further comprising:determining the second representation based at least in part on thefirst representation and a third representation associated with anotherdetected object in the three-dimensional space.
 19. The one or morenon-transitory computer-readable media of claim 16, the operationsfurther comprising: inputting a multi-channel image representing a voxelspace into a machine learning algorithm; and receiving, as the firstrepresentation, an output of the machine learning algorithm, wherein themulti-channel image comprises a length associated with a first dimensionof the three-dimensional space, a width associated with a seconddimension of the three-dimensional space, and a number of channels, andfurther wherein the number of channels is based, at least in part, on athird dimension of the three-dimensional space.
 20. The one or morenon-transitory computer-readable media of claim 19, wherein the numberof channels is further based at least in part on one or more featurescomprising at least one of: an average of sensor data; a covariance ofsensor data; a number of observations of sensor data; occupancy data; orone or more probabilities associated with a semantic classification.